Image processing method, and non-transitory computer-readable storage medium storing image processing program and image processing apparatus

ABSTRACT

The method produces, from a first image, a second image with sparse coding. The method produces, from the first image, a processing intermediate image having a pixel value distribution that a difference among pixel values in a region of the intermediate image is equal to a DC component in a corresponding region of the first image, performs a first process of acquiring, using an AC component in a first region of the intermediate image and a basis produced by dictionary learning, an AC component in a second region, performs a second process of acquiring a difference among pixel values in the second region as a DC component in a corresponding region of the second image, and repeats the first and second processes with changing a position of the first region in the intermediate image to acquire DC components in regions of the second image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing technology fornewly producing an image by sparse coding from a given image.

2. Description of the Related Art

Various image processes are performed using a technology of subtracting,from a pixel value distribution in an arbitrary partial region of aknown image, an average pixel value (DC component) in the partial regionto acquire a component (AC component) and of converting the AC componentinto an AC component in a partial region of an unknown imagecorresponding to the partial region of the known image.

For example, Michael Elad, Michal Aharon, “Image Denoising Via Sparseand Redundant Representations Over Learned Dictionaries”, Transactionson Image Processing, U.S.A., IEEE, 2006, Vol. 15, Issue 12, pp.3736-3745, which is hereinafter referred to as Literature 1, disclosesan image processing method capable of performing a noise removal processwhich produces an original image including no noise from a degradedimage including the noise. Specifically, the method first estimates,from an AC component in a small region (hereinafter, referred to as “anextraction region”) arbitrarily extracted in the degraded image, an ACcomponent including no noise in a small region (hereinafter, referred toas “a corresponding region”) in the original image corresponding to theextraction region. Next, the method adds together a DC component in theextraction region of the degraded image and the estimated AC componentto estimate a pixel value distribution in the corresponding region ofthe original image. The method performs the above processes on an entiredegraded image to produce the original image in which the noise isremoved.

Jianchao Yang, Zhaowen Wang, Zhe Lin, Scott Cohen, Thomas Huang, “CoupleDictionary Training for Image Super-Resolution”, Transactions on ImageProcessing, U.S.A., IEEE, 2012, Vol. 21, Issue 8, pp. 3467-3478, whichis hereinafter referred to as Literature 2, discloses an imageprocessing method capable of performing super-resolution processing ofacquiring, from a low resolution image (degraded image) produced byperforming degradation processing such as decimation of pixels on a highresolution image, a high resolution image equivalent to that before thedegradation processing. Specifically, the method first performsinterpolation processing by a nearest neighbor method or the like on thelow resolution image to produce an intermediate image having a highresolution. Since this intermediate image is smoothed through theinterpolation processing, the method estimates, an AC component in anarbitrary extraction region of the intermediate image, an unsmoothed ACcomponent in a corresponding region of the high resolution image. Next,the method adds together a DC component in the extraction region of theintermediate image and the estimated AC component to estimate a pixelvalue distribution in the corresponding region of the high resolutionimage. The method performs the above processes on an entire intermediateimage to produce a high resolution image subjected to thesuper-resolution processing. The image processing methods disclosed inLiteratures 1 and 2 each use bases previously produced by dictionarylearning from the AC components in multiple small regions extracted fromtraining images before and after their degradation. Such imageprocessing methods are each called “a sparse representation-based imageprocessing method, or “sparse coding” to be used in the followingdescription. The basis is a set of elements as the small regionsproduced by dictionary learning. The training image is an image forproducing the basis by dictionary learning.

The sparse coding disclosed in Literatures 1 and 2 is based on anassumption that the DC component in the extraction region of thedegraded image and the intermediate image which are each an input imageis equal to the DC component in the corresponding region of the originalimage and the high resolution image which are each an output image.Thus, when this assumption holds, the output image can be produced fromthe input image accurately.

However, this assumption does not hold in many cases. For example, thesecases include a case of performing a color conversion of an image of apathological sample stained with a certain color into an image of apathological sample stained with another color and a case ofcalculating, from a sample image of an unknown sample captured through apartially coherent imaging system, a complex amplitude distribution oflight transmitted through the sample. In these cases, the DC componentin the extraction region of the input image differs from the DCcomponent in the corresponding region of the output image, so that thesparse coding disclosed in Literatures 1 and 2 cannot be directlyapplied thereto.

SUMMARY OF THE INVENTION

The present invention provides an image processing method and apparatuscapable of accurately acquiring, from an input image, a DC component inan output image.

The present invention provides as an aspect thereof an image processingmethod of producing, from a first image, a second image by using sparsecoding. The method includes when an average pixel value in a partialregion of an image is referred to as a DC component, and a componentacquired by subtracting the DC component from a pixel value distributionin the partial region is referred to as an AC component: producing, fromthe first image, a processing intermediate image having a pixel valuedistribution in which a difference among multiple pixel values in apartial region of the processing intermediate image is equal to the DCcomponent in a partial region of the first image corresponding to thepartial region of the processing intermediate image; performing a firstprocess of acquiring, by using the AC component in a first partialregion extracted in the processing intermediate image and a basisproduced by dictionary learning, the AC component in a second partialregion; and performing a second process of acquiring a difference amongmultiple pixel values in the second partial region as the DC componentin a partial region of the second image corresponding to the secondpartial region. The method repeats the first and second processes withchanging a position of extracting the first partial region in theprocessing intermediate image to acquire the DC components in multiplepartial regions of the second image.

The present invention provides as another aspect thereof anon-transitory computer-readable storage medium storing an imageprocessing program that causes a computer to execute an image process ofproducing, from a first image, a second image by sparse coding. Theimage process includes when an average pixel value in a partial regionof an image is referred to as a DC component, and a component acquiredby subtracting the DC component from a pixel value distribution in thepartial region is referred to as an AC component: producing, from thefirst image, a processing intermediate image having a pixel valuedistribution in which a difference among multiple pixel values in apartial region of the processing intermediate image is equal to the DCcomponent in a partial region of the first image corresponding to thepartial region of the processing intermediate image; performing a firstprocess of acquiring, by using the AC component in a first partialregion extracted in the processing intermediate image and a basisproduced by dictionary learning, the AC component in a second partialregion; and performing a second process of acquiring a difference amongmultiple pixel values in the second partial region as the DC componentin a partial region of the second image corresponding to the secondpartial region. The image process repeats the first and second processeswith changing a position of extracting the first partial region in theprocessing intermediate image to acquire the DC components in multiplepartial regions of the second image.

The present invention provides as still another aspect thereof an imageprocessing apparatus configured to produce, from a first image, a secondimage by sparse coding. The apparatus includes when an average pixelvalue in a partial region of an image is referred to as a DC component,and a component acquired by subtracting the DC component from a pixelvalue distribution in the partial region is referred to as an ACcomponent: an image producer configured to produce, from the firstimage, a processing intermediate image having a pixel value distributionin which a difference among multiple pixel values in a partial region ofthe processing intermediate image is equal to the DC component in apartial region of the first image corresponding to the partial region ofthe processing intermediate image; a first processor configured toperform a first process of acquiring, by using the AC component in afirst partial region extracted in the processing intermediate image anda basis produced by dictionary learning, the AC component in a secondpartial region; and a second processor configured to perform a secondprocess of acquiring a difference among multiple pixel values in thesecond partial region as the DC component in a partial region of thesecond image corresponding to the second partial region. The imageproducer is further configured to repeat the first and second processeswith changing a position of extracting the first partial region in theprocessing intermediate image to acquire the DC components in multiplepartial regions of the second image.

Other aspects of the present invention will become apparent from thefollowing description and the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of an imageprocessing system of Embodiment 1 of the present invention.

FIG. 2 is a flowchart illustrating a procedure of the image processingmethod in Embodiment 1.

FIG. 3 is a flowchart of Embodiment 2 as an application example of theimage processing method in Embodiment 1.

FIG. 4 is a flowchart of Embodiment 3 as another application example ofthe image processing method in Embodiment 1.

FIG. 5 is a flowchart illustrating a procedure of a complementaryexplanation of Embodiment 3.

FIGS. 6A to 6E illustrate a result of Embodiment 2.

FIGS. 7A to 7C illustrate results obtained through a simple conventionalmethod combination.

FIGS. 8A to 8E illustrate a result of Embodiment 2.

FIGS. 9A to 9C illustrate a result of Embodiment 2.

FIGS. 10A to 10E illustrate a result obtained through the simpleconventional method combination.

FIG. 11 illustrates a relation between an original image and anintermediate image in Embodiment 1.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described belowwith reference to the accompanied drawings.

Embodiment 1

FIG. 1 illustrates a configuration of an image processing system of afirst embodiment (Embodiment 1) of the present invention. This imageprocessing system 100 includes an image processing apparatus 101, animage inputter 102, an image outputter 103 and bus wiring 104. The imageprocessing apparatus 101, the image inputter 102 and the image outputter103 are connected to one another through the bus wiring 104.

The image inputter 102 constituted by a digital camera or a slidescanner and inputs an input image to the image processing apparatus 101.The slide scanner is a pathological sample image acquiring apparatus tobe used for pathological diagnosis. The image inputter 102 may beconstituted by an interface device such as a CD-ROM drive and a USBinterface that read out the input image from a non-transitorycomputer-readable storage medium such as a USB memory and a CD-ROM eachstoring digital image data. The input image is a monochrome image havingtwo-dimensionally arranged data of luminance values, or a color imagehaving two-dimensionally arranged data of luminance values for each ofRGB colors. A color space of the color image is not limited to RGB, andmay be, for example, YCbCr or HSV.

The image outputter 103 is constituted by a display device such as aliquid crystal display and outputs an output image from the imageprocessing apparatus 101. The image outputter 103 may be constituted byan interface device such as a CD-ROM drive and a USB interface and maywrite out the output image to a non-transitory computer-readable storagemedium such as a USB memory and a CD-ROM. The image outputter 103 may beconstituted by a storage apparatus such as a HDD to store the outputimage. The image outputter 103 may be configured to serve as one ofthese three image output apparatuses.

The image processing apparatus 101 is constituted by a computer thatincludes a CPU as a controller and a processor, a RAM as a temporarymemory and a keyboard as an input unit (these are not illustrated). Theimage processing apparatus 101 executes an image process described belowaccording to an image processing program as an installed computerprogram. The image processing apparatus 101 serves as an image producer,a first processor and a second processor.

FIG. 2 is a flowchart illustrating a procedure of an image process(image processing method) performed by the image processing apparatus101. This image process produces, from a first image as a known inputimage, a second image as an unknown output image by using sparse coding.More specifically, the image process acquires a DC component as anaverage pixel value in a partial region (hereinafter, referred to as asmall region) at an arbitrary position in the second image. In thisembodiment, the first image and the second image have an identical sizeto each other. The size of the first and second images and that of thesmall region are expressed by p×q where p represents number of pixels ina vertical direction, and q represents number of pixels in a horizontaldirection. Although this embodiment describes the first image as amonochrome image, processes described below may be performed for eachcolor of, for example, RGB when the first image is a color image.

At step S201, the image processing apparatus 101 prepares (provides) afirst training image and a second training image that are respectivelyused to produce a first basis and a second basis described later. Thefirst training image and the second training image are respectivelyselected for the first image as the input image and the second image asthe output image, in other words, selected as similar images thereto.For example, when a color conversion from an image stained byHematoxylin-Eosin (HE) to an image stained by Direct Fast Scarlet (DFS)is performed, the HE stained image corresponds to the first image, andthe DFS stained image corresponds to the second image. The HE stainedimage and the DFS stained image are pathological sample images of, amongmultiple sections sliced from an identical tissue, two sections slicedat adjacent positions and stained with mutually different colors. Inthis example, the first training image is an image of a pathologicalsample which is an HE stained section of an arbitrary tissue, and thesecond training image is an image of a pathological sample which is aDFS stained section of the tissue. The color conversion in this exampleis a virtual color conversion technique on a computer.

Next, at step S202, the image processing apparatus 101 produces a firsttraining intermediate image from the first training image and produces asecond training intermediate image from the second training image. Thefirst training intermediate image is an image having a pixel valuedistribution in which a difference among multiple pixel values in asmall region of the first training intermediate image is equal to a DCcomponent in a small region of the first training image corresponding tothe small region of the first training intermediate image. Similarly,the second training intermediate image is an image having a pixel valuedistribution in which a difference among multiple pixel values in asmall region of the second training intermediate image is equal to a DCcomponent in a small region of the second training image correspondingto the small region of the second training intermediate image.

This rule for producing the training intermediate image from thetraining image is also applied to producing a processing intermediateimage from the first image which is a processing target image describedlater. Specifically, the processing intermediate image is produced fromthe first image under a rule that the processing intermediate image isan image having a pixel value distribution in which a difference amongmultiple pixel values in a small region of the processing intermediateimage is equal to a DC component in a small region of the first imagecorresponding to the small region of the processing intermediate image.

The training images and the first image (processing target image) arehereinafter collectively referred to as the input image, and thetraining intermediate images and the processing intermediate image arehereinafter collectively referred to as the intermediate image, indetailed description below of the rule.

The term “a small region corresponding to another small region” betweenthe input image and the intermediate image (also between other images)means that these small regions are located at an identical position(coordinates) in these images.

The “multiple pixel values in the small region” of the intermediateimage may be selected as, for example, pixel values at an upper-leftcorner and a lower-right corner of the small region, or pixel values ata lower-left corner and an upper-right corner, or may be selected in anyway. The number of the multiple pixel values is not limited to two andmay be three pixel values or more. When three pixel values or more areselected, a sum of a difference between each pair of pixel values amongthe three pixel values or more corresponds to the “difference” stated inthe rule, and this difference needs to be equal to the DC component inthe corresponding region. For example, when three pixels are selected, asum of a difference between pixel values at the upper-left corner andthe lower-right corner and a difference between pixel values at thelower-left corner and the lower-right corner corresponds to the“difference” stated in the rule.

The small region needs to have a size, that is, numbers of pixels in thevertical direction and the horizontal direction, smaller than those ofthe input image and the intermediate image. The numbers of pixels ineach of these directions needs to be two or more. The size of the smallregion is fixed during the production of the intermediate image from theinput image.

In the following description, a rule represented by followingexpressions (1) and (2) is used as the rule for producing theintermediate image from the input image. First, initial pixel values inthe intermediate image are all set to zero. Next, when a pixel value aat an upper-left corner in a small region at an arbitrary position inthe intermediate image is confirmed to be zero, the pixel value a at theupper-left corner and a pixel value b at a lower-right corner are set asfollows:

$\begin{matrix}{{a = {{- \frac{DC}{2}} + ɛ}}{b = {\frac{DC}{2} + ɛ}}} & (1)\end{matrix}$

where DC represents a DC component in a small region of the input imagecorresponding to a small region of the intermediate image, and εrepresents an arbitrary constant. When the pixel value a at theupper-left corner is not zero, only the pixel value b at the lower-rightcorner in the small region is set as follows:

b=DC+a  (2)

It is obvious from expressions (1) and (2) that a difference among pixelvalues in a small region at an arbitrary position in the intermediateimage is used to calculate a DC component in a small region of anoriginal image corresponding to the small region of the intermediateimage, as follows:

DC=−a+b  (3)

Any rule (for example, a rule that uses a difference among three pixelvalues or more as described above) other than the rule represented byexpressions (1) and (2) may be employed as the rule for producing theintermediate image from the input image. However, in that case,expression (3) needs to be changed according to the employed rule. Thisapplies to other embodiments described later.

FIG. 11 illustrates a relation between expressions (1) to (3) and theinput and intermediate images. A hatched rectangle illustrated on a leftside of FIG. 11 illustrates a DC component in a small region at acertain position in the input image. A rectangle illustrated on a rightside of FIG. 11 is a small region of the intermediate imagecorresponding to the small region of the input image. In the smallregion of the intermediate image, a and b represent the pixel values atthe upper-left and lower-right corners in that small region.

At step S203, the image processing apparatus 101 extracts multiple smallregions from the first training intermediate image and extracts multiplesmall regions from the second training intermediate image respectivelycorresponding to (that is, at positions identical to those of) multiplesmall regions extracted from the first training intermediate image. Theimage processing apparatus 101 sets at random the positions (extractionpositions) of extracting the small regions from the first trainingintermediate image places. Although the extracted small region canpartially overlap with a previously extracted small region, two or moresmall regions fully overlapping with one other cannot be extracted froman identical position. Each small region has a same size as that set atstep S202.

At step S204, the image processing apparatus 101 subtracts, from a pixelvalue distribution in each small region extracted from the firsttraining intermediate image at step S203, a DC component in that smallregion to calculate an AC component therein. The image processingapparatus 101 similarly calculates an AC component in each small regionextracted from the second training intermediate image. Then, the imageprocessing apparatus 101 produces, by using these AC components, by aprocess called dictionary learning, a first basis for the first trainingintermediate image and a second basis for the second trainingintermediate image.

Next, description will be made of a dictionary learning algorithm forproducing the first and second bases. Known dictionary learningalgorithms include joint sparse coding and coupled dictionary learning,which are disclosed in Literature 2. When using the joint sparse coding,the image processing apparatus 101 first converts the AC component inthe small region extracted at a certain position in the first trainingintermediate image into a column vector. Next, the image processingapparatus 101 converts the AC component in the small region of thesecond training intermediate image corresponding to the small regionextracted from the first training intermediate image into a columnvector. Then, the image processing apparatus 101 joints these two columnvectors to produce a vertical column vector. The image processingapparatus 101 performs these processes on the AC components in all theextracted small regions and horizontally joints the produced verticalcolumn vector to produce a matrix.

The image processing apparatus 101 further produces, from this matrix,one basis matrix by using a K-SVD algorithm. The K-SVD algorithm is analgorithm for producing a basis matrix by using a matrix produced from atraining image and is used most popularly in the sparse coding. Althoughthis embodiment employs the K-SVD algorithm in the dictionary learningusing the first and second training intermediate images, any otheralgorithm having a similar functionality may be applicable.

Of the produced basis matrix, an upper half matrix corresponds to thefirst training intermediate image, and a lower half matrix correspondsto the second training intermediate image. The image processingapparatus 101 extracts the upper half matrix from the produced basismatrix, converts the column vectors in that upper half matrix into smallregions and sets a set of the resulting small regions as the firstbasis. The image processing apparatus 101 performs the same conversionon the lower half matrix extracted from the basis matrix and sets a setof the resulting small regions as the second basis. Each small regionhas a same size as that of the small region set at step S202.

In the coupled dictionary learning, learning is performed by a methoddifferent from the above-described method. This embodiment uses thejoint sparse coding and therefore description of the coupled dictionarylearning will be omitted. Either of the joint sparse coding and thecoupled dictionary learning can produce bases for achieving the sameeffects. The number of the small regions, which are elements of thebasis, is previously set by a user.

The processes at steps S201 to S204 do not necessarily need to beperformed by the image processing apparatus 101. That is, the user maypreviously produce the first and second bases using another computer andstore these bases in the image processing apparatus 101. In this case,the image processing apparatus 101 may use the first and second basesthus stored when performing processes at step S205 and subsequent steps.

Next, at step S205, the image processing apparatus 101 produces theprocessing intermediate image from the first image under the ruledescribed in step S202. The first and second training intermediateimages produced from the first and second training images at step S202are used to produce the first and second bases at step S204. On theother hand, the processing intermediate image produced from the firstimage input as the processing target image at this step is used tocalculate a DC component in a small region at an arbitrary position inthe second image as the output image. That is, the image processingapparatus 101 performs image processing on the processing intermediateimage by using the first and second bases calculated at steps S201 toS204 to acquire the DC component in the small region at the arbitraryposition in the second image as the output image.

Next, at step S206, the image processing apparatus 101 extracts a firstsmall region (first partial region) at an arbitrary position in theprocessing intermediate image and approximates an AC component in thefirst small region with a linear combination of the elements of thefirst basis to acquire linear combination coefficients. Theapproximation with the linear combination (hereinafter also referred toas “linear combination approximation”) means that expressing the ACcomponent in the extracted first small region by a weighted sum of smallregions that are the elements of the first basis, and weights of thesmall regions are the linear combination coefficients. Each small regionhas a same size as that set at step S202. The linear combinationapproximation can be represented by following expression (4):

t≈=α ₁ s ₁+α₂ s ₂+ . . . α_(n) s _(n)  (4)

where si (i=1 to n) represents the elements of the first basis, αi (i=1to n) represents a weight for an i-th element of the first basis, thatis, a linear combination coefficient. Moreover, t represents the ACcomponent in the first small region extracted from the processingintermediate image, and n represents number of all the elements of thefirst basis. Algorithms of approximating a small region extracted froman image with a linear combination of elements of a basis includeorthogonal matching pursuit (OMP) disclosed in Literature 1. Althoughthis embodiment uses the OMP to approximate the extracted small regionwith the linear combination of the element of the basis, any otheralgorithm having a similar functionality may be applicable.

At step S207, the image processing apparatus 101 acquires (estimates),by a linear combination of the elements of the second basis with thelinear combination coefficients acquired at step S206, an AC componentin a second small region (second partial region). On an assumption thatthe second image as the output image is produced, an intermediate imagethat is expected to be produced from that second image under the ruledescribed at step S202 is referred to as “a virtual intermediate image”.The second small region at this step is a small region as a virtualregion of the virtual intermediate image corresponding to the firstsmall region extracted from the processing intermediate image at stepS206. The process at this step is a first process.

Next, at step S208, the image processing apparatus 101 acquires, byusing the AC component in the above-mentioned second small region, a DCcomponent in a small region of the second image corresponding to thesecond small region, that is, corresponding to the first small regionextracted from the processing intermediate image. In order to acquirethe DC component in the small region of the second image correspondingto the second small region by using the AC component in the second smallregion, the image processing apparatus 101 uses a reverse procedure tothat for producing the intermediate image from the input image describedat step S202. In other words, the image processing apparatus 101calculates a difference between the pixel values at the upper-leftcorner and the lower-right corner in the second small region to acquirethe DC component in the small region of the second image correspondingto the second small region. The process at this step is a secondprocess.

When any rule other than expressions (1) and (2) is used for producingthe intermediate image from the input image, the method of acquiring theDC component in the small region of the second image corresponding tothe second small region from the AC component in that second smallregion needs to be changed according to the rule to be used.

Next, description will be made of grounds for enabling the acquisitionof the DC component in the small region of the second imagecorresponding to the second small region from a difference amongmultiple pixel values in the second small region. As described at stepS207, the AC component in the second small region is the AC componentestimated in the small region of the virtual intermediate image for thesecond image. As shown by expression (3), using the difference betweenthe pixel values at the upper-left corner and the lower-right corner ina small region (hereinafter, referred to as “an intermediate smallregion”) at an arbitrary position in the intermediate image enablesproviding a DC component in a small region of the input imagecorresponding to the intermediate small region. The pixel values at theupper-left corner and the lower-right corner in the intermediate smallregion are sums of a common DC component and different AC componentsfrom each other. Thus, the difference between both the pixel values is adifference between the AC components, and this difference is the DCcomponent in the small region of the second image corresponding to theintermediate small region. This is expressed by following expression(5):

$\begin{matrix}\begin{matrix}{{DC} = {{- a} + b}} \\{= {{- \left( {\overset{\sim}{a} + m} \right)} + \left( {\overset{\sim}{b} + m} \right)}} \\{= {{- \overset{\sim}{a}} + \overset{\sim}{b}}}\end{matrix} & (5)\end{matrix}$

where a and b represent the pixel values at the upper-left corner andthe lower-right corner (sums of the DC component and the AC componentthereat) in a small region (referred to as “an intermediate smallregion”) in the intermediate image, m represents the DC component in theintermediate small region, and a and b with “˜” respectively representvalues of the AC components at an upper-left corner and a lower-rightcorner in the intermediate small region. Moreover, DC represents the DCcomponent in the small region of the second image corresponding to theintermediate small region.

The image processing apparatus 101 thus acquiring the DC component inone small region of the second image proceeds to step S209.

At step S209, the image processing apparatus 101 repeats, until the DCcomponents in the small regions at all different positions in the secondimage are acquired, the processes at steps S206 to S208 with changingthe position of extracting the first small region at step S206. In thecase of allowing the extraction of the partially overlapped smallregions from the processing intermediate image at step S206, thatextraction may also cause overlap in a finally acquired DC component ina small region at a certain position in the second image. In that case,at the position where the overlap in the DC component in the smallregion of the second image is caused, the image processing apparatus 101adds together the overlapped DC components and divides the additionresult by number of the overlapped DC components at each pixel toacquire the DC components in the small regions at all positions in thesecond image. When the acquisition of the DC components in all the smallregions in the second image completes, the image processing apparatus101 completes the production of the second image and thus ends theprocess.

Although the described process at step S209 acquires the DC componentsin the small regions at the all positions in the second image, the DCcomponents in the small regions at the all positions are not necessarilyneeded to be acquired. That is, only the DC components in small regions(partial region of the second image) at multiple positions depending onan intended use of the second image may be acquired.

The procedure described above enables acquiring, from the known firstimage, the DC component in the small region at any arbitrary position inthe unknown second image or in a partial region thereof.

Next, description will be made of application examples of the imageprocessing described in Embodiment 1.

Embodiment 2

Description will be made of an image processing method of performing acolor conversion from a known first image to an unknown second image,which is a second embodiment (Embodiment 2) of the present invention,with reference to a flowchart in FIG. 3. In this embodiment, the firstimage and the second image have an identical size to each other. Thefirst image is an input image (processing target image), and the secondimage is an output image. Although this embodiment also describes thecase where the first image is a monochrome image, when the first imageis a color image in a color space such as RGB, YCbCr and HSV, thefollowing processes may be performed thereon for each color the firstimage is a color image of a color space such as RGB, YCbCr and HSV, asdescribed in Embodiment 1.

The flowchart in FIG. 3 illustrates a procedure of a color conversionprocess (image processing method) performed by the image processingapparatus 101. The image processing apparatus 101 as a computer executesthe color conversion process described below according to a colorconversion program (image processing program) as an installed computerprogram.

At step S301, the image processing apparatus 101 prepares (provides)first and second training images. These training images are prepared ina same manner as that described at step S201 of Embodiment 1 (FIG. 3).

Next, at step S302, the image processing apparatus 101 extracts multiplesmall regions from the first training image. Moreover, the imageprocessing apparatus 101 extracts, from the second training image,multiple small regions corresponding to the multiple small regions (thatis, located at identical positions) of the first training image. Eachsmall region has a size smaller than that of the first and second imagesand has each side of two pixels or more. In this embodiment, the size ofthe small region is 8×9 pixels. The small region is extracted under asame rule as that described at step S203 in Embodiment 1.

Next, at step S303, the image processing apparatus 101 produces, byusing the method described at step S204 in Embodiment 1, from ACcomponents in the small regions extracted at step S302, a first ACcomponent basis and a second AC component basis which are used foracquiring AC components in the second image. The first and second ACcomponent bases are hereinafter respectively abbreviated as a first ACbasis and a second AC basis. Numbers of elements of the first and secondAC basis are desirable to be large. However, a larger number of theelements needs a longer calculation time for producing the first andsecond AC bases. For this reason, in this embodiment, the numbers of theelements of the first and second AC bases are each set to 1024.

The processes at steps S301 to S303 do not necessarily need to beperformed by the image processing apparatus 101. That is, a user maypreviously produce the first and second bases using another computer andstore these bases in the image processing apparatus 101. In this case,the image processing apparatus 101 may use the first and second basesthus stored when performing processes at step S304 and subsequent steps.

At step S304, the image processing apparatus 101 extracts a small regionat an arbitrary position in the first image and approximates an ACcomponent in the extracted small region with a linear combination of theelements of the first AC basis to acquire linear combinationcoefficients. The linear combination approximation is performed asdescribed at step S206 in Embodiment 1. The extracted small region has asame size as that set at step S302.

Next, at step S305, the image processing apparatus 101 estimates, by alinear combination of the elements of the second AC basis with thelinear combination coefficients acquired at step S304, an AC componentin a small region of the second image corresponding to the small regionextracted from the first image. The processes at steps S304 and S305correspond to a third process.

Next, at step S306, the image processing apparatus 101 repeats, untilthe AC components in the small regions at all different positions in thesecond image are acquired, the processes at steps S304 and S305 withchanging the position of extracting the small region at step S304. Afteracquiring the AC components in the small regions at all the positions inthe second image, the image processing apparatus 101 proceeds to stepS307.

At step S307, the image processing apparatus 101 adds together the ACcomponent in each of the small regions of the second image acquired atsteps S301 to S306 and the DC components in same each small regionacquired at steps S201 to S209 described in Embodiment 1 to produces thesecond image. When the DC component in the small region is acquired asdescribed at steps S201 to S209, the size of the small region is set tothe size in this embodiment at step S202. In addition, in a case wherepartial overlap of the small regions extracted from the first image isallowed at step S304, the same process as that described at step S209 inEmbodiment 1 is performed.

The procedure described above enables performing the color conversionfrom the known first image to the unknown second image.

Description will be made of an example of a color conversion from the HEstained image (first image) to the DFS stained image (second image)through the color conversion process in this embodiment. FIG. 6Aillustrates the HE stained image. FIG. 6B illustrates the DFS stainedimage acquired from the HE stained image by the color conversion. FIG.6C illustrates a ground truth DFS stained image. FIGS. 6D and 6Erespectively illustrate the first basis and the second basis.

All the images are normalized such that a sum of squares of pixel valuesin each image becomes 1. Each of all the images has a size of 120×160pixels. FIGS. 6D and 6E illustrate the first and second bases in each ofwhich 32 elements each including 8×9 pixels are tiled in each ofvertical and horizontal directions.

Evaluation of a similarity between the ground truth DFS stained imageand the DFS stained image acquired from the HE stained image by usingroot mean square error (RMSE) resulted in an RMSE of 6.6528E-4. The RMSEis a square root of a value obtained by dividing a sum of squares of adifference of pixel values between an evaluation target image and areference image by number of pixels in the reference image. Theevaluation target image and the reference image have an identical sizeto each other. In this embodiment, the reference image is the groundtruth DFS stained image, and the evaluation target image is the acquiredDFS stained image. Simply put, a smaller RMSE indicates a highersimilarity of the acquired DFS stained image to the ground truth DFSstained image.

Next, description will be made of, in order to show superiority of theimage process of this embodiment to conventional image processes, anexample of a color conversion from the HE stained image to the DFSstained image by using a combination of a method called “integral image”and the sparse coding. Although the integral image and the sparse codingare each well known by itself, their combination has not been reported.

A simple combination of these methods would hardly acquire an accurateDC component for a reason described later. This simple combination ofthese methods is referred to as “a simple conventional methodcombination” to be distinguished from the image processing method ofthis embodiment. The integral image is a method of producing anintermediate image from an input image under a rule represented byfollowing expression (6):

I(x,y)=i(x,y)+I(x−1,y)+I(x,y−1)−I(x−1,y−1)  (6)

where i(x, y) represents a pixel value at coordinates (x, y) of theinput image, and I(x, y) represents a pixel value at coordinates (x, y)of the intermediate image. As a specific method of applying the rulerepresented by expression (6), scanning of the coordinates (x, y) isperformed in the intermediate image from a pixel at an upper-left cornertoward a right direction in a pixel line to sequentially output I(x, y)calculated by expression (6). When the scanning reaches a rightmostpixel of the pixel line, the scanning is repeated from a leftmost pixelto a rightmost pixel in a pixel line lower by one pixel than the scannedpixel line. This method produces the intermediate image from the inputimage. In the specific method, initial pixel values of the intermediateimage are all zero. The input image and the intermediate image have anidentical size to each other. When expression (6) needs information ofpixel values such as I(0, 1) and I(1, 0) outside the intermediate image,those pixel values are set to zero.

The intermediate image thus produced enables acquiring, from pixelvalues at four corners of a small region (intermediate small region) atan arbitrary position in the intermediate image, a sum of pixel valuesin a small region of the input image corresponding to the intermediatesmall region by following expression (7):

V=I1+I4−(I2+I3)  (7)

where I1, I2, I3 and I4 represent the pixel values at the four corners(that is, pixel values at the upper-left corner, the upper-right corner,the lower-left corner and the lower-right corner) of the intermediatesmall region at the arbitrary position. In addition, V represents thesum of the pixel values in the small region of the input imagecorresponding to the intermediate small region.

However, as a characteristic of the integral image, a correct sum of thepixel values in the small region (hereinafter also referred to as “acorresponding small image”) of the input image corresponding to theintermediate small region is not always acquired. To be precise, the sumV calculated by expression (7) is a sum of pixel values in a smallregion other than first row and first column pixels of the correspondingsmall region of the input image. Thus, using the sum V of the pixelvalues calculated by expression (7) does not enable providing a correctDC component in the corresponding small region of the input image.However, this embodiment uses, in order to use the integral image, asthe DC component in the corresponding small region of the input image, avalue given by following expression (8):

$\begin{matrix}{{DC} = \frac{V}{N}} & (8)\end{matrix}$

where V represents the sum of the pixel values in the correspondingsmall region of the input image which is calculated by expression (7),and DC represents the DC component in the corresponding small region ofthe input image. In addition, N represents number of pixels other thanthe first row and first column pixels of the corresponding small regionof the input image.

FIGS. 7A to 7C illustrate an example in which this integral image wasused at step S202 for performing the color conversion from the HEstained image to the DFS stained image by the simple conventional methodcombination.

This embodiment produces the intermediate image from the input imageunder the rule represented by expressions (1) and (2), whereas thesimple conventional method combination produced the intermediate imagefrom the input image under the rule represented by expression (6). Sincethe intermediate image is produced from the input image under thedifferent rules from each other, the method of calculating the DCcomponent in the small region of the second image corresponding to thesecond small region from the AC component in the second small region atstep S208 is different between in this embodiment and in the simpleconventional method combination. Accordingly, this embodiment uses therule represented by expression (3), whereas the simple conventionalmethod combination used the rule represented by expressions (7) and (8).

The processes at other steps S201, S203 to S207, S209 and S301 to S307were performed similarly to this embodiment, and thereby the colorconversion from the HE stained image to the DFS stained image wasperformed by the simple conventional method combination.

FIG. 7A illustrates the DFS stained image acquired from the HE stainedimage by the color conversion with the simple conventional methodcombination. FIGS. 7B and 7C respectively illustrate a first basis and asecond basis produced by the simple conventional method combination. Theacquired DFS stained image is normalized such that a sum of squares ofpixel values becomes 1. The acquired DFS stained image has a size of120×160 pixels. FIGS. 7B and 7C illustrate the first and second bases ineach of which 32 elements each including 8×9 pixels are tiled in each ofvertical and horizontal directions.

Evaluation of the similarity between the ground truth DFS stained imageillustrated in FIG. 6C and the DFS stained image acquired by the simpleconventional method combination resulted in an RMSE of 1.4603E-3. Thisproves that the DFS stained image acquired in this embodiment (its RMSEis 6.6528E-4) is more similar to the ground truth DFS stained image thanthat acquired by the simple conventional method combination (its RMSE is1.4603E-3). This is because this embodiment and the simple conventionalmethod combination use mutually different rules for producing theintermediate image from the input image. Although the simpleconventional method combination uses the conventional rule representedby expression (6), this embodiment uses the rule represented byexpressions (1) and (2) which is a unique rule to this embodiment. Asdescribed above, as the characteristic of the integral image, usingexpression (7) cannot provide a correct DC component in thecorresponding small region of the input image. In contrast, thisembodiment employing expression (3) enables providing an accurate DCcomponent in the corresponding small region of the input image. That is,this embodiment can provide a more accurate solution than that of thesimple conventional method combination. Since the produced intermediateimages are different between in this embodiment and in the simpleconventional method combination, the bases produced by the dictionarylearning using the intermediate image (training intermediate image) arealso different between in them.

Embodiment 3

In a third embodiment (Embodiment 3) of the present invention,description will be made a method of calculating, from a sample imageacquired by image capturing of an unknown sample through a partiallycoherent or completely coherent imaging system, a complex amplitudedistribution of light transmitted through the sample, with reference toa flowchart illustrated in FIG. 4. The sample image corresponds to thefirst image (input image) in Embodiment 1, and an image having thecomplex amplitude distribution corresponds to the second image (outputimage). To be precise, the complex amplitude distribution of the lighttransmitted through the sample means phase and amplitude distributionsof an electric field of the light transmitted through the sample. Thecomplex amplitude distribution of the light transmitted through thesample is hereinafter simply referred to as “a complex amplitudedistribution of the sample”.

An image showing the complex amplitude distribution of the sample isconstituted by an image of a real part (hereinafter also referred to as“a real-part image”) and an image of an imaginary part (hereinafter alsoreferred to as “an imaginary-part image”). The sample image, thereal-part image and the imaginary-part image have an identical size toone another. The imaginary-part image is two-dimensionally arranged dataof real numbers except an imaginary unit i.

The partially coherent imaging system means an imaging optical system inwhich lights transmitted through two arbitrary points of the sampleinterfere with each other depending on their phases and illuminationconditions, such as a bright field microscope. The completely coherentimaging system means an imaging optical system in which lightstransmitted through two arbitrary points of the sample interfere witheach other depending only on their phases, which is achieved by, forexample, illumination with laser light.

Although this embodiment describes the first image as a monochromeimage, as described in Embodiment 1, when the first image is a colorimage of a color space such as RGB, YCbCr and HSV, the followingprocesses may be performed for each color.

The flowchart in FIG. 4 illustrates a procedure of a sample imageprocess (image processing method) performed by the image processingapparatus 101. The image processing apparatus 101 executes the sampleimage process described below according to a sample image processingprogram as an installed computer program.

At step S401, the image processing apparatus 101 provides (prepares) areal-part image and an imaginary-part image that show a complexamplitude distribution of a training sample and provides a trainingimage acquired by image capturing of the training sample through thepartially coherent or completely coherent imaging system. The complexamplitude distribution of the training sample can be modeled, when thesample is, for example, a section of a tissue obtained from a patient,by using known refractive indices of constituent elements (such as anucleus and a cell cytoplasm) of a cell. The complex amplitudedistribution may be acquired by using data obtained through a digitalholography microscope or the like. When using the former, the imageprocessing apparatus 101 produces the training image by image capturingsimulation. When using the latter, the image processing apparatus 101produces the training image by the image capturing simulation or anactual image capturing through the microscope.

Next, at step S402, the image processing apparatus 101 extracts multiplesmall regions from the training image. Moreover, the image processingapparatus 101 also extracts, from each of the real-part andimaginary-part images of the complex amplitude distribution of thetraining sample, multiple small regions at corresponding positions tothose of the small regions extracted from the training image. Each smallregion needs to have a size smaller than that of the sample image and tohave each side of two pixels or more. In this embodiment, the size ofthe small region is 6×8 pixels. The small region is extracted under thesame rule as that described at step S203 in Embodiment 1.

Next, at step S403, the image processing apparatus 101 produces, byusing the method described at step S204, from AC components in the smallregions extracted at step S402 from the training image, the real-partimage and the imaginary-part image, a first AC component basis, a secondAC component basis and a third AC component basis (hereinafterrespectively referred to as “a first AC basis”, “a second AC basis” and“a third AC basis”). Although the description of step S204 was made ofthe method of extracting the small regions from two images andperforming dictionary learning on the AC components in the extractedsmall regions to produce two bases, the method is also applicable tothis case of three images.

Next, description will be made of a method of producing the first,second and third AC bases. The image processing apparatus 101 firstextracts a small region at a certain position in the training image.Similarly, the image processing apparatus 101 extracts, from each of thereal-part and imaginary-part images of the complex amplitudedistribution of the training sample, a small region corresponding to thesmall region extracted from the training image. Then, the imageprocessing apparatus 101 converts these three extracted small regionsinto column vectors and vertically joints these three column vectors toproduce a long column vector. The image processing apparatus 101 repeatsthis process on all the small regions extracted from the training imageand from the real-part and imaginary-part images of the complexamplitude distribution of the training sample to produce a matrix inwhich the long column vectors are horizontally jointed.

The image processing apparatus 101 then produces one basis matrix fromthe produced matrix by the K-SVD algorithm. Of the produced basismatrix, a top third part corresponds to the training image, a bottomthird part corresponds to the imaginary-part image of the trainingsample, and a middle third part corresponds to the real-part image ofthe training sample. The image processing apparatus 101 then extracts,from the basis matrix, the parts respectively corresponding to thetraining image, the real-part image and the imaginary-part image andconverts the column vectors of each of the extracted parts into smallregions. Sets of these small regions converted from the training,real-part and imaginary-part images are respectively set as the first,second and third AC bases. For the same reason as that described inEmbodiment 2, number of elements of each of the first, second and thirdAC bases is set to 1024.

The processes at steps S401 to S403 do not necessarily need to beperformed by the image processing apparatus 101. That is, a user maypreviously produce the first to third bases using another computer andstore these bases in the image processing apparatus 101. In this case,the image processing apparatus 101 may use the first to third bases thusstored when performing processes at step S404 and subsequent steps.

Next, at step S404, the image processing apparatus 101 extracts a smallregion at an arbitrary position in the sample image and approximates anAC component in the extracted small region (hereinafter, referred to asthe sample image small region) with a linear combination of the elementsof the first AC basis to acquire linear combination coefficients. Thelinear combination approximation is performed as described at step S206in Embodiment 1. The sample image small region has a same size as thatset at step S402.

Next, at step S405, the image processing apparatus 101 estimates, by alinear combination of the elements of the second AC basis with thelinear combination coefficients acquired at step S404, an AC componentin a small region of the real-part image (of the complex amplitudedistribution of the unknown sample) corresponding to the sample imagesmall region. Similarly, the image processing apparatus 101 estimates,by a linear combination of the elements of the elements of the third ACbasis with the linear combination coefficients acquired at step S404, anAC component in a small region of the imaginary-part image (of thecomplex amplitude distribution of the unknown sample) corresponding tothe sample image small region. The processes at steps S404 and S405 area third process.

Next, at step S406, the image processing apparatus 101 repeats, untilthe AC components in the small regions at all different positions ineach of the real-part and imaginary-part images of the complex amplitudedistribution of the sample are acquired, the processes at steps S404 andS405 with changing the position of extracting the sample image smallregion at step S404. After acquiring the AC components in the smallregions at all the positions in the real-part and imaginary-part images,the image processing apparatus 101 proceeds to step S407.

At step S407, the image processing apparatus 101 adds together, at eachof all the positions in the real-part image of the complex amplitudedistribution of the unknown sample, the AC components in the smallregion acquired at steps S404 to S406 and the DC component thereinacquired at steps S201 to S209 described in Embodiment 1. This processprovides the real-part image of the complex amplitude distribution ofthe sample. Moreover, the image processing apparatus 101 acquires, in asimilar manner, the imaginary-part image of the complex amplitudedistribution of the sample.

In addition, in a case where partial overlap of the small regionsextracted from the sample image is allowed at S404, the same process asthat described at step S209 in Embodiment 1 is performed.

The procedure described above enables acquiring the complex amplitudedistribution of the unknown sample from the sample image.

Next, description will be made of acquiring at step S407, from thesample image, by using the processes at steps S201 to S209, the DCcomponent in a small region at an arbitrary position in the complexamplitude distribution of the sample, with reference to a flowchart inFIG. 5.

At step S501, the image processing apparatus 101 prepares (provides)training images. As the prepared training images, the real-part andimaginary-part images of the complex amplitude distribution of thetraining sample and the training image which are prepared at step S401can be used.

Next, at step S502, the image processing apparatus 101 produces, by themethod described at step S202, a first training intermediate image fromthe training image. Similarly, the image processing apparatus 101produces a second training intermediate image from the real-part imageof the complex amplitude distribution of the training sample andproduces a third training intermediate image from the imaginary-partimage of the complex amplitude distribution of the training sample.

Next, at step S503, the image processing apparatus 101 extracts, by themethod described at step S203, multiple small regions from the firsttraining intermediate image. The image processing apparatus 101 alsoextracts, from each of the second and third training intermediateimages, multiple small regions corresponding to the small regionsextracted from the first training intermediate image. The extractedsmall regions have a same size as that set at step S402.

Next, at step S504, the image processing apparatus 101 produces, by themethod described at step S403, a first basis, a second basis and a thirdbasis from AC components in the small regions extracted at step S503.

The processes at steps S501 to S504 do not necessarily need to beperformed by the image processing apparatus 101. That is, a user maypreviously produce the first to third bases using another computer andstore these bases in the image processing apparatus 101. In this case,the image processing apparatus 101 may use the first to third bases thusstored when performing processes at step S505 and subsequent steps.

Next, at step S505, the image processing apparatus 101 produces, by themethod described at step S202, a processing intermediate image from thesample image.

Next, at step S506, the image processing apparatus 101 extracts a firstsmall region at an arbitrary position in the processing intermediateimage and approximates an AC component in the extracted first smallregion with a linear combination of elements of the first basis toacquire linear combination coefficients. The linear combinationapproximation is performed as described at step S206 in Embodiment 1.The extracted small region has a same size as that set at step S503.

Next, at step S507, the image processing apparatus 101 estimates, by alinear combination of elements of the second basis with the linearcombination coefficients acquired at step S506, an AC component in asecond real-part small region described below. On an assumption that thereal-part image of the complex amplitude distribution of the sample asan output image is produced, an intermediate image that is expected tobe produced from that real-part image under the rule described at stepS202 in Embodiment 1 is referred to as “a virtual real-part intermediateimage”. The second real-part small region used in this step is a smallregion as a virtual region of this virtual real-part intermediate imagecorresponding to the first small region of the processing intermediateimage extracted at step S506.

Moreover, the image processing apparatus 101 estimates, by a linearcombination of elements of the third basis with the linear combinationcoefficients acquired at step S506, an AC component in a secondimaginary-part small region described below. On an assumption that theimaginary-part image of the complex amplitude distribution of the sampleas an output image is produced, an intermediate image that is expectedto be produced from that imaginary-part image under the rule describedat step S202 in Embodiment 1 is referred to as “a virtual imaginary-partintermediate image”. The second imaginary-part small region used in thisstep is a small region as a virtual region of this virtualimaginary-part intermediate image corresponding to the first smallregion of the processing intermediate image extracted at step S506. Theprocess at this step is a first process.

Next, at step S508, the image processing apparatus 101 acquires, fromthe second real-part small region, a DC component in a small region ofthe real-part image (of the complex amplitude distribution of thesample) corresponding to the second real-part small region. The DCcomponent is acquired by a same method as that described at step S208 inEmbodiment 1.

Furthermore, the image processing apparatus 101 similarly acquires, fromthe second imaginary-part small region, a DC component in a small regionof the imaginary-part image (of the complex amplitude distribution ofthe sample) corresponding to the second imaginary-part small region. Theprocess at this step is a second process.

Next, at step S509, the image processing apparatus 101 repeats, untilthe DC components in the small regions at all different positions ineach of the real-part and imaginary-part images of the complex amplitudedistribution of the sample are acquired, the processes at steps S506 toS508 with changing the position of extracting the first small regions atstep S506. In a case where partial overlap of the small regionsextracted from the processing intermediate image is allowed at stepS506, the same process as that described at step S209 in Embodiment 1 isperformed.

The procedure described above enables acquiring, from the sample image,the DC component in the small region at any arbitrary position in eachof the real-part and imaginary-part images of the complex amplitudedistribution of the unknown sample.

FIGS. 8A to 8E and 9A to 9C illustrate an example of acquisition of thecomplex amplitude distribution of the unknown sample from the sampleimage produced by image capturing of the sample through the partiallycoherent imaging system. FIG. 8A illustrates the sample image acquiredby capturing of an optical image of the unknown sample formed by thepartially coherent imaging system. FIG. 8B illustrates the real-partimage of the complex amplitude distribution of the sample acquired fromthe sample image by the method described in this embodiment. FIG. 8Cillustrates the imaginary-part image of the complex amplitudedistribution of the sample acquired from the sample image by the methoddescribed in this embodiment. FIG. 8D illustrates a ground truthreal-part image of a complex amplitude distribution of the sample. FIG.8E illustrates a ground truth imaginary-part image of the complexamplitude distribution of the sample.

FIG. 9A illustrates the first basis, FIG. 9B illustrates the secondbasis, and FIG. 9C illustrates the third basis.

All the images are normalized such that a sum of squares of pixel valuesin each image becomes 1. The partially coherent imaging system used toacquire the sample image has the following optical conditions. Anobject-side numerical aperture of an imaging lens is 0.7, an inner σ ofan annular light source is 0.3, an outer σ thereof is 0.7, and awavelength of illumination light is 0.55 μm. Symbol σ represents a ratioof a numerical aperture of an illumination optical system and theobject-side numerical aperture of the imaging lens. In this example,with a maximum optical path length difference generated when theillumination light is transmitted through the sample being assumed to be2.88 radian, the complex amplitude distribution of the sample wasproduced through simulation. This is equivalent to modeling of a cellwith a maximum refractive index difference of 0.05 between a nucleus anda cell cytoplasm. These values are exemplary used in this embodiment,and other values may be used. All images have a size of 200×200 pixels.

FIGS. 9A to 9C illustrate the first, second and third bases in each ofwhich 32 elements each including 6×8 pixels are tiled in each ofvertical and horizontal directions.

Table 1 collectively shows results of evaluation of similarities withRMSE between the real-part image and the imaginary-part image of thecomplex amplitude distribution of the sample acquired from the sampleimage and the ground truth real-part image and the imaginary-part imageof the complex amplitude distribution of the sample.

TABLE 1 RMSE between real-part image ot complex amplitude 5.4135E−03distribution of sample acquired from sample image and ground truthreal-part image of complex amplitude distribution of sample RMSE betweenimaginary-part image of complex amplitude 8.1681E−04 distribution ofsample acquired from sample image and ground truth imaginary-part imageof complex amplitude distribution of sample

Next, similarly to Embodiment 2, in order to show superiority of thisembodiment, description will be made of an example of acquisition of acomplex amplitude distribution of an unknown sample acquired from asample image produced using the partially coherent imaging system, bythe above-described simple conventional method combination in which theintegral image and the sparse coding are combined. First, the integralimage was used at step S502, and thereby the complex amplitudedistribution of the unknown sample from the sample image was acquired bythe simple conventional method combination. In other words, instead ofthe rule represented by expressions (1) and (2) used in this embodiment,the rule represented by expression (6) described in Embodiment 2 wasused to produce the intermediate image from the input image. In thesimple conventional method combination, since the intermediate image isproduced from the input image by using the different rule from that inthis embodiment as described above, the DC component is acquired fromthe AC component in the second small region at step S508 by a differentmethod from that in this embodiment. Accordingly, instead of the methodrepresented by expression (3) used in this embodiment, the methodrepresented by expressions (7) and (8) described in Embodiment 2 wasused.

The processes at other steps S501, S503 to S507, S509 and S401 to S407were performed similarly to this embodiment, and thereby the complexamplitude distribution of the sample was acquired from the sample imageby the simple conventional method combination.

FIGS. 10A to 10E illustrate an example of acquisition of the complexamplitude distribution of the unknown sample acquired from the sampleimage by the simple conventional method combination. FIG. 10Aillustrates a real-part image of the complex amplitude distribution ofthe sample acquired from the sample image by the simple conventionalmethod combination. FIG. 10B illustrates an imaginary-part image of thecomplex amplitude distribution of the sample acquired from the sampleimage by the simple conventional method combination. FIGS. 10C, 10D and10E respectively illustrate a first basis, a second basis and a thirdbasis produced by the simple conventional method combination. All theimages are normalized such that a sum of squares of pixel values in eachimage becomes 1. Each of the images has a size of 200×200 pixels. FIGS.10C to 10E illustrate the first, second and third bases in each of which32 elements each including 6×8 pixels are tiled in each of vertical andhorizontal directions.

Table 2 collectively shows results of evaluation of similarities withRMSE between the real-part image and the imaginary-part image of thecomplex amplitude distribution of the sample acquired from the sampleimage by the simple conventional method combination and the ground truthreal-part image and the imaginary-part image of the complex amplitudedistribution of the sample illustrated in FIGS. 8D and 8E.

TABLE 2 RMSE between real-part image of complex amplitude 6.5216E−03distribution of sample acquired fiom sample image and ground truthreal-part image of complex amplitude distribution of sample RMSE betweenimaginary-part image of complex amplitude 6.8089E−03 distribution ofsample acquired from sample image and ground truth imaginary-part imageof complex amplitude distribution of sample

As understood from a comparison between Table 2 and Table 1, thereal-part and imaginary-part images acquired in this embodiment are moresimilar to the ground truth real-part and imaginary-part images of thesample than those acquired by the simple conventional methodcombination. This is because the rules for producing the intermediateimage from the input image are different between this embodiment and thesimple conventional method combination as described above, which resultsin different bases produced by dictionary learning with the producedintermediate image.

Each of the above embodiments can acquire accurate DC components inpartial regions of the second image as the output image from the firstimage as the input image.

Therefore, using each of the above embodiments enables performing acolor conversion from an image of a pathological sample stained with acolor to an image stained with another color and enables acquiring acomplex amplitude distribution of light transmitted through an unknownsample from a sample image produced by image capturing of the samplethrough a partially or completely coherent imaging system.

OTHER EMBODIMENTS

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions recorded on a storage medium (e.g., non-transitorycomputer-readable storage medium) to perform the functions of one ormore of the above-described embodiment (s) of the present invention, andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment (s). The computer may comprise one or more ofa central processing unit (CPU), micro processing unit (MPU), or othercircuitry, and may include a network of separate computers or separatecomputer processors. The computer executable instructions may beprovided to the computer, for example, from a network or the storagemedium. The storage medium may include, for example, one or more of ahard disk, a random-access memory (RAM), a read only memory (ROM), astorage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2013-263173, filed on Dec. 20, 2013, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing method of producing, from afirst image, a second image by using sparse coding, the methodcomprising when an average pixel value in a partial region of an imageis referred to as a DC component, and a component acquired bysubtracting the DC component from a pixel value distribution in thepartial region is referred to as an AC component: producing, from thefirst image, a processing intermediate image having a pixel valuedistribution in which a difference among multiple pixel values in apartial region of the processing intermediate image is equal to the DCcomponent in a partial region of the first image corresponding to thepartial region of the processing intermediate image; performing a firstprocess of acquiring, by using the AC component in a first partialregion extracted in the processing intermediate image and a basisproduced by dictionary learning, the AC component in a second partialregion; and performing a second process of acquiring a difference amongmultiple pixel values in the second partial region as the DC componentin a partial region of the second image corresponding to the secondpartial region, wherein the method repeats the first and secondprocesses with changing a position of extracting the first partialregion in the processing intermediate image to acquire the DC componentsin multiple partial regions of the second image.
 2. An image processingmethod according to claim 1, further comprising for producing the basis:providing training images for the respective first and second images;producing, from each of the training images, a training intermediateimage having a pixel value distribution in which a difference amongmultiple pixel values in a partial region of the training intermediateimage is equal to the DC component in a partial region of the trainingimage corresponding to the partial region of the training intermediateimage; and producing the basis by dictionary learning using the ACcomponents in multiple partial regions extracted from the trainingintermediate image.
 3. An image processing method according to claim 1further comprising performing a third process of acquiring, by using theAC component in the partial region extracted in the first image and anAC component basis produced by dictionary learning, the AC component inthe partial region of the second image corresponding to the partialregion extracted in the first image, wherein the method repeats thethird process with changing a position of extracting the partial regionin the first image to acquire the AC components in multiple partialregions in the second image, and the method adds together the ACcomponents acquired in the multiple partial regions of the second imageand the DC components acquired therein in each of the correspondingpartial regions to produce the second image.
 4. An image processingmethod according to claim 1 of acquiring, as the first image, a sampleimage produced by imaging of a sample through a partially coherent orcompletely coherent imaging system and of producing, from the sampleimage, the second image showing a complex amplitude distribution oflight transmitted through the sample, wherein: the method furthercomprises performing a third process of acquiring, by using the ACcomponent in a partial region extracted in the sample image and an ACcomponent basis produced by dictionary learning, the AC components inpartial regions of a real-part image and an imaginary-part image whichare images of a real part and an imaginary part of the complex amplitudedistribution of the sample, the partial regions of the real-part andimaginary-part images corresponding to the partial region extracted inthe sample image, and the method repeats the third process with changinga position of extracting the partial region in the sample image tocalculate the AC component in each of multiple partial regions in thereal-part image and in the imaginary-part image, and the method addstogether the AC components acquired in the multiple partial regions ofeach of the real-part image and the imaginary-part image and the DCcomponents acquired therein in each of the corresponding partial regionsto acquire the complex amplitude distribution.
 5. An image processingmethod according to claim 3 further comprising for producing the ACcomponent basis: providing training images for the first and secondimages or for the sample image and the sample; and producing the ACcomponent basis by dictionary learning using the AC components inmultiple partial regions extracted from the training images.
 6. Anon-transitory computer-readable storage medium storing an imageprocessing program that causes a computer to execute an image process ofproducing, from a first image, a second image by sparse coding, theimage process comprising when an average pixel value in a partial regionof an image is referred to as a DC component, and a component acquiredby subtracting the DC component from a pixel value distribution in thepartial region is referred to as an AC component: producing, from thefirst image, a processing intermediate image having a pixel valuedistribution in which a difference among multiple pixel values in apartial region of the processing intermediate image is equal to the DCcomponent in a partial region of the first image corresponding to thepartial region of the processing intermediate image; performing a firstprocess of acquiring, by using the AC component in a first partialregion extracted in the processing intermediate image and a basisproduced by dictionary learning, the AC component in a second partialregion; and performing a second process of acquiring a difference amongmultiple pixel values in the second partial region as the DC componentin a partial region of the second image corresponding to the secondpartial region, wherein the image process repeats the first and secondprocesses with changing a position of extracting the first partialregion in the processing intermediate image to acquire the DC componentsin multiple partial regions of the second image.
 7. An image processingapparatus configured to produce, from a first image, a second image bysparse coding, the apparatus comprising when an average pixel value in apartial region of an image is referred to as a DC component, and acomponent acquired by subtracting the DC component from a pixel valuedistribution in the partial region is referred to as an AC component: animage producer configured to produce, from the first image, a processingintermediate image having a pixel value distribution in which adifference among multiple pixel values in a partial region of theprocessing intermediate image is equal to the DC component in a partialregion of the first image corresponding to the partial region of theprocessing intermediate image; a first processor configured to perform afirst process of acquiring, by using the AC component in a first partialregion extracted in the processing intermediate image and a basisproduced by dictionary learning, the AC component in a second partialregion; and a second processor configured to perform a second process ofacquiring a difference among multiple pixel values in the second partialregion as the DC component in a partial region of the second imagecorresponding to the second partial region, wherein the image produceris further configured to repeat the first and second processes withchanging a position of extracting the first partial region in theprocessing intermediate image to acquire the DC components in multiplepartial regions of the second image.